Multiagent reinforcement learning: algorithm converging to Nash equilibrium in general-sum discounted stochastic games

Reinforcement learning turned out a technique that allowed robots to ride a bicycle, computers to play backgammon on the level of human world masters and solve such complicated tasks of high dimensionality as elevator dispatching. Can it come to rescue in the next generation of challenging problems like playing football or bidding on virtual markets? Reinforcement learning that provides a way of programming agents without specifying how the task is to be achieved could be again of use here but the convergence of reinforcement learning algorithms to optimal policies is only guaranteed under the conditions of stationarity of the environment that is violated in multiagent systems. For reinforcement learning in multiagent environments general-sum discounted stochastic games become a formal framework instead of Markov decision processes. Also the optimal polCite as: Multiagent Reinforcement Learning Algorithm Converging to Nash Equilibrium, Natalia Akchurina,Proc. of 8th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2009), Decker, Sichman, Sierra and Castelfranchi (eds.), May, 10–15, 2009, Budapest, Hungary, pp. XXX-XXX. Copyright c © 2009, International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved. icy concept in multiagent systems is different — we can’t speak anymore about optimal policy (policy that provides the maximum cumulative reward) without taking into account the policies of other agents that influence our payoffs. In the environment where every agent tries to maximize its cumulative reward it is the most natural to accept Nash equilibrium as the optimal solution concept. In Nash equilibrium each agent’s policy is the best response to the other agents’ policies. Thus no agent can gain from unilateral deviation. A number of algorithms [1, 2, 4, 5, 6, 7, 8, 9] were proposed to extend reinforcement learning approach to multiagent systems. The convergence to Nash equilibria was proved for very restricted class of environments: strictly competitive [6], strictly cooperative [2, 7] and 2-agent 2action iterative game [1]. In [5] convergence to Nash equilibrium has been achieved in self-play for strictly competitive and strictly cooperative games under additional very restrictive condition that all equilibria encountered during learning stage are unique [7]. In this paper we propose a reinforcement learning algorithm that converges to Nash equilibria with some given accuracy in general-sum discounted stochastic games and prove it formally under some assumptions. We claim that it is the first algorithm that finds Nash equilibrium for the general case. The paper is organized as follows. In section 2 we present formal definitions of stochastic game, Nash equilibrium, as well as prove some theorems that we will need for equilibrium approximation theorem in section 3. Section 3 is devoted to discussion and necessary experimental estimations of the conditions of the equilibrium approximation theorem. In sections 5 and 6 the developed algorithm Nash-DE and analysis of the results of experiments are presented correspondingly.